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User interface 



The invention relates to a system comprising: 
display means for displaying a target; and 

user input means for accepting a touch-less user input of a user who is 
pointing to the target. 

5 

An embodiment of an interactive system which is arranged to accept user 
input of a user who is pointing to a target is well-known. Typically, the target corresponds to 
a button of a graphical user interface being displayed by a display device. Various 
10 technologies exist to perform touch sensing, i.e. accepting the user input by sensing the touch 
of the user: 

Capacitive: A tiny current is fed in to a conductive coating within a glass 
sandwich from bus bars at the edge of the glass. The current leaks to ground through a finger 
and electronics determine an x-y position in relation to each of the bus bars; 
1 5 - Resistive: A flexible layer with conductive coating on one side is brought into 

contact with a rigid layer to form a circuit with a varying resistance. An x-y position is 
measured by volt drop from each of 4 bus bars at the edge; and 

Surface-Acoustic Wave: Exciters around the edge of a layer of glass set up 
vibrations on its surface. Touching the glass causes a shadow which is detected by receivers 
20 also around the edge. A system based on this technology is typically also arranged to detect 
pressure giving a z-axis reading. 

The above-mentioned technologies need the finger or other object actually 
touching the screen. Technologies that do not need an actual contact are also well known. 
Interactive systems which are based on these technologies are arranged to accept a touch-less 
25 user input. Examples of these technologies are: 

Projected Capacitive: Fine wires embedded in a glass substrate project an 
electrical field on the surface of the glass layer which is disturbed by finger presence. The x-y 
position is determined by electronics built into the sensor; 
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Infrared: Infrared (IR) technology relies on the interruption of an IR light grid 
in front of the screen of the display device. The touch frame or opto-matrix frame contains a 
row of IR-light emitting diode (LEDs) and photo transistors, each mounted on two opposite 
sides to create a grid of invisible infrared light. The frame assembly is comprised of printed 
5 wiring boards on which the opto-electronics are mounted and is concealed behind an IR- 
transparent bezel. The bezel shields the opto-electronics from the operating environment 
while allowing the IR beams to pass through. The IR controller sequentially pulses the LEDs 
to create a grid of IR light beams. When a stylus, such as a finger, enters the grid, it obstructs 
the beams. One or more photo transistors detect the absence of light and transmits a signal 

10 that identifies the x and y coordinates; 

Charge Transfer Sensing: By charging a sense electrode, which can be 
anything electrically conductive, to a fixed potential, then transferring that charge to a charge 
detector comprising another known capacitor, the capacitance of the sense electrode can be 
readily ascertained. The charge and transfer operations are conducted by switches. This is 

1 5 ideally done by digital logic; in fact, the sensor is almost ideally suited to digital control and 
processing from start to finish. The only analog signal is a typically slow signal requiring no 
special precautions. Conversion of this to digital can be performed by any of a number of 
commercially available ADC chips. 

Cross Capacitance Sensing: See for instance the article "3D Touchless Display 

20 Interaction", by C. van Berkel, in SID Proceedings International Symposium, vol. 33, number 
2, ppl410-1413, 2002". Capacitive sensors are arranged around the edge of a display to 
detect hand and finger positions. Implemented with either transistors on glass or with 
conventional CMOS, this offers a better front-of-screen performance and lower cost than 
conventional touch screens. 

25 An advantage of touch-less user input means is that in theory contamination 

can be prevented. Contamination can be prevented in both direction: the display means 
displaying the "button" can not get dirty and the finger that pressed the "button" cannot get 
contaminated. With "button" is meant a visualization of a target to be pointed at. 

Applications that are likely taking advantage of the first situation are for 

30 instance kitchen appliances, display in the kitchen (turn page while kneading the dough), in 
the bathroom, in a car garage (present cars need a PC for maintenance and grease). For the 
second situation an operating room is a good example: the surgeons can keep there hand 
sterile while controlling the medical equipment. In general these buttons are very hygienic, 
antiseptic and stay germfree. 
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3 

Although, touch-less user input means do exist, it appears that very often the 
contamination of is not prevented. In other words, although there is no actual need for 
touching e.g. a screen in order to provide user input, many users do touch the screen and 
hence the screen or finger get dirty. 

5 

It is an object of the invention to provide a system of the kind described in the 
opening paragraph which is arranged to minimize the probability that a user touches the 
system. 

1 0 This object of the invention is achieved in that the system further comprises 

feedback means for providing a tactile feedback to the user in response of the touch-less user 
input. The above-mentioned problem arises from the natural tendency to touch a "button" to 
activate it. So, even if the system reacts visibly for instance because a finger is detected by 
the user input means in front of the display means, users often keep moving until they touch 

1 5 something. By generating something which is to be sensed by the user, in particular by the 
finger or hand with which the user is providing the user input, the user receives an 
appropriate feedback which is a sign that the system has accepted the user input. This tactile 
feedback will prevent the user to continue with its movement towards the display means and 
to touch the display means. 

20 Although the display means might be based on a print, i.e. showing a static 

picture, it is preferred that the display means according to the invention comprises a display 
device for displaying a state dependent graphical user interface comprising the target. With a 
display device is meant a device having a structure of light generating elements which are 
independently controllable to generate different images subsequently. For example a CRT 

25 (Cathode Ray Tube), LCD (Liquid Crystal Display) or PDP (Plasma display panel). 

In an embodiment of the system according to the invention, the feedback 
means is arranged to eject a fluid in a predetermined direction which is related to the touch- 
less user input. Preferably, the predetermined direction is such that the fluid touches the 
finger or hand directly after the user input has been accepted. Experiments have shown that 

30 by ejecting a fluid with a velocity which exceeds a predetermined threshold in such a 

predetermined direction, an appropriate feedback can be provided to the user. The fluid can 
be any type of material which does not result in an effect which takes longer than e.g. one 
second. A fluid can be a liquid or a gas. 
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Optionally, the fluid has a temperature which is some degrees higher or lower 
than room temperature. The temperature difference enhances the touch sensation which is 
caused by the speed of the ejected fluid, i.e. the pressure of the ejected fluid as perceived by 
the user. 

5 Preferably, the fluid comprises air. An advantage of applying air is that no 

separate container is required which has to be refilled with a special fluid. 

In another embodiment of the system according to the invention, the feedback 
means comprises a heating device for generating heath to be noticed by the user. By locally 
generating heath the user is provided with a heath sensation. Preferably, the heat is provided 

10 with a pulse which does not take long, e.g. less than 100 milliseconds. 

It will be clear that other types of feedback which can be sensed by e.g. the 
fingers or hand, i.e. the part of the body of the user with which the user input is provided to 
the system, can be applied. Even a spark or electrical shock with an intensity which is below 
a predetermined threshold, i.e. pain level, can be applied to provide the feedback. 

15 In an embodiment of the system according to the invention, the user input 

means comprises means for sensing capacitance related to the user. An advantage of such a 
technology is that it is relatively robust. 

In an other embodiment of the system according to the invention, the user 
input means comprises a light source which is arranged to cooperate with a light sensor. An 

20 advantage of such technology for accepting user input is that it is relatively cheap. 

An embodiment of the system according to the invention further comprises 
audio generating means for generating audio feedback. Besides generating tactile feedback it 
is advantageous to provide additional feedback by means of an audio signal which is 
generated synchronously with the tactile feedback. 

25 In an embodiment of the system according to the invention, the display means 

is arranged to display the target a predetermined distance in front of a screen of the display 
means. With in front of the screen is meant that the target is closer to the user then the screen. 
In standard displays, i.e. not according to the invention, a target is projected at the screen of 
the display device or even a predetermined distance behind the screen. In the latter case the 

30 screen is e.g. a plate of glass which is disposed in front of the luminance and/or color 

modulation and/or generation unit. For instance in the case of an LCD display a plate of glass 
is disposed in front of the polarization layer. 

Typically, a user wants to touch the target. By displaying the target in a 
surface which is located at a predetermined distance in front of the screen, the user perceives 
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visual feedback when he reaches e.g. with his finger to that surface, i.e. when he crosses the 
surface in which the target is projected. So by displaying the target a predetermined distance 
in front of the screen, the risk of contamination is further reduced. 

In an embodiment of the system according to the invention, the display means 
5 is arranged to display the target with a first appearance corresponding to a first state and to 
display the target with a second appearance in response of the touch-less user input. In other 
words, visible feedback is generated too, by means of updating the appearance of the 
graphical user interface. It is well-known in the art of graphical user interfaces to indicate that 
e.g. a button is "pushed", or that a scroll bar is selected for movement. 

10 

These and other aspects of the system according to the invention will become 
apparent from and will be elucidated with respect to the implementations and embodiments 
described hereinafter and with reference to the accompanying drawings, wherein: 
15 Fig. 1 schematically shows an embodiment of the system according to the 

invention; 

Fig. 2 schematically shows another embodiment of the system according to the 
invention, comprising display means for displaying the target a predetermined distance in 
front of the screen; 

20 Fig. 3 schematically shows another embodiment of the system according to the 

invention, comprising heat generating means; 

Fig. 4A schematically shows another embodiment of the system according to 
the invention, whereby the display means comprises a mirror; 

Fig. 4B schematically shows the inside of a car, with an embodiment of the 
25 system according to the invention being located in the dash-board of the car; and 

Figs. 5 A and 5B schematically show the concept of cross capacitance sensing. 
Same reference numerals are used to denote similar parts throughout the figures. 

30 Fig. 1 schematically shows a first embodiment of the system 100 according to 

the invention. The system 100 comprises: 

a display device 1 02 for displaying targets e.g. 112, 116; 

user input means 104-1 10, 1 14 for accepting a touch-less user input of a user 
101 who is pointing to one of the targets 1 16; and 
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feedback means 1 17-122 for providing a tactile feedback to the user 101 in 
response of the touch- less user input. 

The targets corresponds to user interface gadgets of a graphical user interface. 
Fig. 1 depicts twenty of these gadgets showing the respective characters "0"-"9" and "a"-"j". 
5 These user interface gadgets corresponds to respective functions of a particular application 
which runs on the system 100. The display device 102 is arranged to display the graphical 
components of the user interface of the system 100. The display device 102 comprises e.g. a 
CRT, LCD or PDP. 

The user input means comprises a number of sensing means 104-1 10, 1 14 

10 being disposed adjacent to the display device 102. The sensing of the sensing means 104-1 10, 
1 14 might be based on acoustic, infra-red or radio frequency waves. The sensing means are 
arranged to detect the presence of an object in their vicinity. If e.g. one of the sensors 104 
which is disposed at the left side of the display device 102 detects the presence of an object, 
e.g. a finger of a user, then the application which runs on the system 1 00 interprets that as 

15 user input. For instance a function which is related to target 1 12 is activated. Sensing means 
based on light are commercially available, e.g. at IVO GmbH & Co. Typically these optical 
sensing means are arranged to measure a distance. These sensing means are so-called 
proximity detectors. The appliance of these sensing means in the system according to the 
invention is as follows. If e.g. a hand is put in the neighborhood of the display device 102 in 

20 order to provide the system with user input then a particular one of the sensors detects that 
there is an object close to it. So, each of the sensors measures the distance of objects in their 
respective neighborhoods. If one of the measured distances is below a predetermined 
threshold then a user input is accepted. Sensing means based on ultra-sound are also 
commercially available. 

25 Alternative user input means might be applied in the system according to the 

invention. Instead of proximity sensors pairs of light generating and light sensing elements 
being disposed in a regular structure can be applied. The presence of an object like a finger of 
a user results in an obstruction of light going from a light generating element to the 
corresponding light sensing element. A further alternative is based on capacity sensing. This 

30 is explained in more detail in connection with Fig. 5A and 5B. A further alternative is based 
on imaging the user by means of cameras. Cameras in combination with a suitable 
programmed image processing unit are advantages for accepting touch-less user input. 

The feedback means comprises a number of tubes, e.g. 122 with a number of 
equidistantly disposed nozzles 1 17, 120. The nozzles 1 17, 120 comprise respective valves 
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1 15, 1 18 with which the nozzles can be closed. The tubes are connected to a pump for 
providing fluid under a predetermined pressure. The pump is not depicted in Fig. 1. The 
location of the nozzles is related to the location of the respective targets. As can be seen in 
Fig. 1, there is a nozzle for each of the targets. 

The working of the system 100 according to the invention is as follows. If the 
system 100 is turned on, a graphical user interface is displayed on the display device 102. 
Suppose that a user wants to control the application which runs on the system 100. In 
particular, to user wants to start the function corresponding to the target indicated with 
reference number 1 16. In order to start the invention he points at that particular target 1 16, 
i.e. that means that he moves his finger in the direction of the location of the display device 
102 where that particular target 1 16 is displayed. As soon as the distance between the finger 
and the display device 102 is below a predetermined distance the presence of the finger is 
observed by two sets of sensing means, i.e. a first light generating element 1 10 in cooperation 
with a first light sensing element 108 and a second light generating element 1 14 in 
cooperation with a second light sensing element 1 13. The presence of the finger is interpreted 
as user input. It should be noted that the user does not have to touch the display device 102. 
The user input is processed by the system. As a result the pump is activated, the valve 1 1 5 is 
opened and consequently a puff of gas leaves the appropriate nozzle 1 17. The user perceives 
a little pressure and hence is notified that his user input has been accepted by the system 100. 

It will be clear that although Fig. 1 shows a nozzle for each of the targets, 
other configurations are possible. For instance a single nozzle might be applied in connection 
with multiple targets or vice versa. 

Generating the tactile feedback by means of ejection of a fluid, preferably a 
puff of air can be realized by means of several types of pressure generating means. It might 
be based on a compressor in conjunction with a container in which the pressure is relatively 
high compared with the atmosphere. Alternatively the pressure pulse is generated 
momentously. 

A puff of air can be generated relatively simple by means of a loudspeaker that 
is blocked except for a small hole. A low frequency signal which is provided at the terminals 
of the loudspeaker will generate a flow of air. A small dedicated cylinder with a piston on a 
permanent magnet forms the basis of an alternative implementation. The permanent magnet 
cylinder is surrounded by a solenoid electro-magnet. A short pulse current will push the 
magnet out and thereby actuating the piston in the cylinder. The produced puff of air is 
clearly noticeable. Further improvements are possible. 
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Fig. 2 schematically shows a second embodiment of the system 200 according 
to the invention, comprising display means 102 for displaying a target 1 16 in a surface 204 
which is located at a predetermined distance in front of the screen 202. There are several 
techniques to achieve this, i.e. to create a so-called virtual display or target. A relatively 
5 simple construction is based on a single lens or is based on a set of parabolic mirrors. An 
alternative construction for displaying targets at such a surface 204 is disclosed in US patent 
6,492, 961. Displaying the targets at such a surface 204 is an important measure to prevent 
touching the screen 202 of the display device 102. This is a feed- forward mechanism to stop 
the movement of the hand of the user at the position of the visible image of the target and not 

1 0 to continue up to the screen 202 of the display device 1 02 itself. 

Besides displaying the target 1 1 6 at a predetermined distance from the display 
device 102, the system 200 as depicted in Fig. 2 comprises further measures to prevent 
touching the screen. First of all, the system 200 comprises tactile feedback means as 
described in connection with Fig. 1. That means a nozzle 120 for ejecting a fluid. The system 

1 5 200 further comprises a loudspeaker 206 for providing audio feedback in response to the user 
input. The application running on the system 200 is arranged to change the appearance of the 
target 1 16 as a result of the user input. The system 200 comprises user input means for 
accepting the user input. These user input means are not depicted in Fig. 2. 

Fig. 3 schematically shows a third embodiment of the system 300 according to 

20 the invention, comprising heat generating means. This system 300 according to the invention 
basically comprises the same components as the system 200 as described in connection with 
Fig. 2. The difference between the two systems is related to the implementation of the tactile 
feedback means. The system 300 comprises a heat generating element 302 which is arranged 
to produce warmth having an energy level which can be well perceived by a user without 

25 hurting the user. Preferably, the heat generating element 302 is arranged to radiate the 
warmth 304 primarily in a direction towards the finger or hand of the user and preferably 
only during a relatively short time interval. Otherwise the system 300 according to the 
invention would dissipate too much energy without any use. 

Fig. 4A schematically shows the system 400 according to the invention 

30 whereby the display means comprises a mirror. The display means comprises a display 

device 102 in front of which a semi-transparent mirror is located. The behavior of the display 
means is as follows. If the display device 102 is turned off, then the user will see a reflection 
of his own face and body. However if the display device 102 is turned on, then the user will 
primarily see the images being generated by means of the display device 102. The system 
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400 as depicted in Fig. 4A comprises a second display device which is disposed at the border 
of the mirror and which is applied to display a graphical user interface comprising the targets 
corresponding to buttons showing the characters "V", "A", "S" and "P". The tactile feedback 
means, e.g. nozzles being arranged to eject air, are integrated in the frame 404 of the mirror. 
5 It will be clear that a system 400 having a mirror as screen should be kept clean. Hence, it is 
advantageous to apply the invention in such a in interactive display system 400. 

Fig. 4B schematically shows the inside of a car, with the system 402 according 
to the invention being located in the dash-board of the car. Fig. 4B shows the view 424 on the 
road in front of the car, the steering-wheel 422 of the driver and a mirror 426. The mirror 426 

10 is optional. Alternatively the display system 402 provides a view to the driver of the car 

which corresponds to images being captured by a camera which is located such that the scene 
behind the car can be imaged. The system 402 comprises sensing means 104 as described in 
connection with Fig 1 or 5 A and 5B. Alternatively, the system 402 comprises cameras 106a- 
106b being located in the ceiling and/or wall of the car. These cameras 106a- 106b are 

15 designed to observe the environment in front of the display device 102. If one of the cameras 
106a- 106b images an arm, hand or fingers, the orientation of it is determined. The 
determined orientation is used to establish to which one of the targets the user is pointing at. 

Figs. 5A and 5B schematically show the concept of cross capacitance sensing. 
Fig. 5A shows a simple combination of two electrodes 502-504 forming a capacitor, with a 

20 capacitive current flowing between them. Fig. 5B shows the same two electrodes 502-504 
whereby a hand 101 is placed near the electrodes 502-504. Some of the field lines 506-510 
are terminated on the hand and the current is decreased relative to the current which flows in 
the case without the hand 101 being placed near the electrodes, like in Fig 5 A. A measurable 
effect can be obtained in which the spatial range is roughly equivalent to the separation 

25 between the electrodes 502-504. Arrays of multiple electrodes offer rich possibilities for 
object and gesture recognition. For instance, n transmitters and n receivers arranged around 
the edge of a display contain n 2 electrode pairs at many different separations. See for more 
details the article "3D Touchless Display Interaction", by C. van Berkel, in SID Proceedings 
International Symposium, vol. 33, number 2, ppl410-1413, 2002. 

30 It should be noted that the above-mentioned embodiments illustrate rather than 

limit the invention and that those skilled in the art will be able to design alternative 
embodiments without departing from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be constructed as limiting the claim. 
The word 'comprising' does not exclude the presence of elements or steps not listed in a 
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claim. The word "a" or "an" preceding an element does not exclude the presence of a 
plurality of such elements. The invention can be implemented by means of hardware 
comprising several distinct elements and by means of a suitable programmed computer. In 
the unit claims enumerating several means, several of these means can be embodied by one 
5 and the same item of hardware. The usage of the words first, second and third, etcetera do not 
indicate any ordering. These words are to be interpreted as names. 
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1 . A system (1 00) comprising: 

display means (1 02) for displaying a target; 

user input means ( 1 04- 1 1 0) for accepting a touch-less user input of a user 
(101) who is pointing to the target (1 16); and 
5 - feedback means (120) for providing a tactile feedback to the user (101) in 

response of the touch-less user input. 

2. A system (100) as claimed in Claim 1 , wherein the display means (102) 
comprises a display device for displaying a graphical user interface comprising the target 

10 (116). 

3. A system (100) as claimed in Claim 1 or 2, wherein the feedback means is 
arranged to eject a fluid in a predetermined direction which is related to the touch-less user 
input. 

15 

4. A system (100) as claimed in Claim 3, wherein the fluid comprises air. 

5. A system (300) as claimed in Claim 1 or 2, wherein the feedback means 
comprises a heating device for generating heath to be noticed by the user (101). 

20 

6. A system (100) as claimed in Claim 1, wherein the user input means 
comprises means for sensing capacitance related to the user (101). 

7. A system (100) as claimed in Claim 1, wherein the user input means (104-1 10) 
25 comprises a light source which is arranged to cooperate with a light sensor. 



8. A system (200, 300) as claimed in Claim 1 , further comprising audio 

generating means (206) for generating audio feedback. 
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9. A system (200, 300) as claimed in Claim 2, wherein the display means (102) is 

arranged to display the target (1 16) a predetermined distance in front of a screen (202) of the 
display means (102). 



5 



10. A system (100) as claimed in Claim 2, wherein the display means (102) is 

arranged to display the target (116) with a first appearance corresponding to a first state and 
to display the target (116) with a second appearance in response of the touch-less user input. 
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